from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.969990	0.148898	NaN	...	brute	-1	1	0.663	0.344495	0.004538	1.000	5.718491	5.718987	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.022266	0.002180	NaN	...	brute	-1	1	1.000	19.071551	0.004902	0.757	0.001168	0.001168	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.921108	0.028251	NaN	...	brute	-1	5	0.757	19.200300	0.068813	0.882	0.152139	0.152140	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.210127	0.010942	NaN	...	brute	1	100	0.882	0.347571	0.006628	1.000	6.358773	6.359929	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.020651	0.000483	NaN	...	brute	1	100	1.000	19.115524	0.065219	0.757	0.001080	0.001080	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.907368	0.040011	NaN	...	brute	-1	100	0.882	19.357950	0.011190	0.663	0.150190	0.150190	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.184117	0.008282	NaN	...	brute	1	5	0.757	0.271847	0.006676	1.000	8.034358	8.036780	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.020806	0.000499	NaN	...	brute	1	5	1.000	4.041141	0.026332	0.922	0.005149	0.005149	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.191104	0.003320	NaN	...	brute	1	1	0.663	4.135509	0.014379	0.929	0.288019	0.288020	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.730465	0.035290	NaN	...	brute	-1	1	0.896	0.272538	0.005492	1.000	6.349440	6.350729	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.005685	0.003775	NaN	...	brute	-1	1	1.000	4.057441	0.018263	0.922	0.001401	0.001401	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.772486	0.027425	NaN	...	brute	-1	5	0.922	4.124163	0.018803	0.896	0.672254	0.672261	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.953	0.0	-1	1	19.378	0.165	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.891	0.0	-1	5	0.347	0.008	1.000	0.033	0.033	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.903	0.0	1	100	19.717	0.051	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.851	0.0	-1	100	0.353	0.007	1.000	0.033	0.033	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.892	0.0	1	5	4.133	0.020	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.0	6.900	0.0	1	1	0.275	0.004	1.000	0.042	0.042	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.0	0.330	0.0	-1	1	4.116	0.026	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.0	0.326	0.0	-1	5	0.272	0.005	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.970	0.149	0.000	0.002	-1	1	0.344	0.005	1.000	5.718	5.719	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.002	0.000	0.022	-1	1	19.072	0.005	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.921	0.028	0.000	0.003	-1	5	19.200	0.069	0.882	0.152	0.152	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	5	0.354	0.004	1.000	0.067	0.067	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.210	0.011	0.000	0.002	1	100	0.348	0.007	1.000	6.359	6.360	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	100	19.116	0.065	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.907	0.040	0.000	0.003	-1	100	19.358	0.011	0.663	0.150	0.150	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.003	0.000	0.024	-1	100	0.345	0.002	1.000	0.070	0.070	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.184	0.008	0.000	0.002	1	5	0.272	0.007	1.000	8.034	8.037	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	5	4.041	0.026	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.191	0.003	0.001	0.001	1	1	4.136	0.014	0.929	0.288	0.288	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	1	0.273	0.005	1.000	0.071	0.071	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.730	0.035	0.000	0.002	-1	1	0.273	0.005	1.000	6.349	6.351	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	0.000	0.006	-1	1	4.057	0.018	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.772	0.027	0.000	0.003	-1	5	4.124	0.019	0.896	0.672	0.672	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.003	0.000	0.007	-1	5	0.272	0.004	1.000	0.026	0.026	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.787703	0.941130	NaN	...	kd_tree	-1	1	0.929	2.964274	0.212304	1.000	0.265732	0.266413	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002757	0.000535	NaN	...	kd_tree	-1	1	1.000	118.957929	0.000000	0.946	0.000023	0.000023	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.068765	0.363824	NaN	...	kd_tree	-1	5	0.946	118.046898	0.000000	0.951	0.009054	0.009054	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.509321	0.446831	NaN	...	kd_tree	1	100	0.951	2.958515	0.164419	1.000	1.862192	1.865065	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003461	0.001023	NaN	...	kd_tree	1	100	1.000	117.529886	0.000000	0.946	0.000029	0.000029	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.285155	0.279967	NaN	...	kd_tree	-1	100	0.951	119.269927	0.000000	0.929	0.027544	0.027544	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.701067	0.310033	NaN	...	kd_tree	1	5	0.946	0.006120	0.000125	1.000	277.944902	278.002669	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001737	0.000487	NaN	...	kd_tree	1	5	1.000	0.047488	0.010557	0.911	0.036583	0.037476	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.934033	0.304402	NaN	...	kd_tree	1	1	0.929	0.068268	0.001213	0.894	13.681901	13.684059	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.027836	0.012823	NaN	...	kd_tree	-1	1	0.891	0.006608	0.001085	1.000	4.212503	4.268875	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.002797	0.000275	NaN	...	kd_tree	-1	1	1.000	0.043076	0.000353	0.911	0.064930	0.064932	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.024767	0.001410	NaN	...	kd_tree	-1	5	0.911	0.042778	0.000343	0.891	0.578966	0.578984	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.205	0.027	0.025	0.0	-1	1	119.539	0.000	0.929	0.027	0.027	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.899	0.040	0.021	0.0	-1	5	2.966	0.191	1.000	1.314	1.317	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.940	0.064	0.020	0.0	1	100	119.691	0.000	0.951	0.033	0.033	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.893	0.059	0.021	0.0	-1	100	2.942	0.146	1.000	1.323	1.325	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.848	0.074	0.021	0.0	1	5	0.043	0.001	0.891	89.883	89.892	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.901	0.050	0.021	0.0	1	1	0.006	0.000	1.000	682.433	682.546	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.017	0.0	-1	1	0.068	0.001	0.894	0.013	0.013	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.025	0.0	-1	5	0.006	0.000	1.000	0.112	0.112	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.788	0.941	0.000	0.001	-1	1	2.964	0.212	1.000	0.266	0.266	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	118.958	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.069	0.364	0.000	0.001	-1	5	118.047	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.941	0.158	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.509	0.447	0.000	0.006	1	100	2.959	0.164	1.000	1.862	1.865	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	117.530	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.285	0.280	0.000	0.003	-1	100	119.270	0.000	0.929	0.028	0.028	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.922	0.159	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.701	0.310	0.000	0.002	1	5	0.006	0.000	1.000	277.945	278.003	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.047	0.011	0.911	0.037	0.037	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.934	0.304	0.000	0.001	1	1	0.068	0.001	0.894	13.682	13.684	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.006	0.000	1.000	0.195	0.195	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.028	0.013	0.001	0.000	-1	1	0.007	0.001	1.000	4.213	4.269	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	1	0.043	0.000	0.911	0.065	0.065	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	0.001	0.000	-1	5	0.043	0.000	0.891	0.579	0.579	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.006	0.000	1.000	0.400	0.400	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.123708	0.002074	300	...	100	100	10	0.824	0.445694	0.0113	1.0	0.277562	0.277651	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	104.934	0.0	300	0.001	0.001	0.524	0.029	0.824	200.067	200.373	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.124	0.002	300	0.006	0.0	0.446	0.011	1.0	0.278	0.278	See	See